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PREFACE 



The Special rnterest. Group (3IG) On Researah Management of the Americ 
aducatlonal Reaearoh Association was Initially conceived by members of the 
Association to "provide a locus within ASBA for activities intended to foster 
more effectlvo and appropriate application of research results in educational 
practice". The SIC has the following basic objectives j 

1. To prQmote nieanlngful discourse about the management 
of reRearch among tlie members of the Special Interest 
Group on Hosearch Management. (The term "research" 
iRcludes research, development, Qvaluatlon, dlssemina- 
tion of Itnowledge, diffusion of developments, and 
related activities.) 

2, To promotQ disaenilnatlon activities designed to Improve 
the management of educational resQarch, 

3, To encourage the acieatiflc study of the managemoQt at 
all administrative levels of educational rcsoarch efforts. 

4. To identify and clarify issues and processes related 
to the selection and training of educational research 
managers , 

Among its activities, tlie 3IG sponsors files of research and research 
related publications, meetings at national, regional, state and local levels, 
issues periodic publications, acts as a elearlng house of information and 
oonduots liaison with counterpart groups and committees of other professional 
societies and associatloQS, 
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INTRODUCTIOiS' 

Don B. Croft and Adolph J. sCoenig 

The Speoial Interest GrouiJ for research management presonted a 
symposium on Applications of Research Slanagement Techniques at the 
annual meeting of the American Educational Research AsBociatlon In 
New York City, February 6, 1971. Panel membors for the symposium 
were selected from organizations conducting research and development, 
iiach panel member presented a paper describing specific appUcatione of 
research management techniques, and then responded to questions from 
members of the audiencB. 

The thrust of the symposium was upon applications in order to 
provide symposium participants with a description of research manf.gement 
techniques currently being used by E&D organizations. Although the 
syr.,iposlum was not specifically designed to include all aspects of resonrch 
management, the palmers when taken together comprise a "working" model 
for managing the information associated with ri&2 activities. 

It is important to note that the topics of the papers were selected by 
people working in R&n organizations. Practicing R&D managers present- 
ed topics that they had found important in the management of research. 
Pressing needs In an organization provide the impetus for the development 
of systematic ways of managing Information. Accordingly, the omergence 
of specific research management techniques carries with it the implicit 
assumption that the organization planed a high priority on a particular 
aspect of rerierach management. Now, a brief resume of the salient 
aspects of each paper is presented. 

The prototyplc reserach management system described by Don 1;. 
Croft includes techniques for organizing and managing information about, • 
1) projects currently being conducted by the organization, 2) reports and 
products completed by the organization, 3) the background and experience 
of members of the organization, 4) procedures for preparing proposals 
forB&D projects. 5) reporting the activities of the organization, and 
6) securing evaluative Informatioti about R&D reports. 

Croft identifies specific types of information for describing the 
activities of RSiD organizations, and presents techniques for orgaiizlng 
the Information so that It may be easily retrieved by research managers. 
I hus, the use of this prototyplc system would allow research managers to 
meet many of the requests for descriptive Information about the organiza- 
tion from members of tlis public or the educational profession. 
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Dan Woclpert discusses a topic which, for tte most part, has been 
igQored by tho E&D community. The topic, organlzatloQal operations, 
described the ^ocedures and policies that reduce the admioiatrative 
burdQn on researchers and developers and assist raaQagemeQt in operat- 
ing an effeptlve R&D organization. Specifically, "An Operations Manual 
for Research Organizations", describes the policies and internal operat- 
ing proeeduros of the Wisoonsin Ran Center, and includes the standing 
operating procedm-ea for the following topics' 1) Contractual Commit- 
ments, 2) GoyerQlng and advisory groups, 3) Personnel Policies, 
4) Program Planning and Budgeting, 5) Toohnioal Development Programs, 
6) Di ssemlnation Programs, and T;, Operatioos and Business Activities. 
In brief, the Woolpert handbook opens to view an operational set of gaiides 
for establishing standardized procedures for conducting R &D activities, 

John Vinsonhaler proposes a system for using the computer to 
manage the information about the financial expenditures of an E&D organ- 
ization. The paper also describes the value of management Information 
systems (MIS) for the administration of scientific research projects. The 
scope presented included. 1) description of an MIS technique contalolng a 
praeUcal method of providing Information on university research projects, 
2) a case study describing the use of an Inexpensleve oomputerizGd MIS 
to manage the flnanoes of an eduo'^.tlonal researoh project and 3) comments 
on the value of the MI3 approach in other areai of researoh management 
supporting appUcations such as literature searching, data collection, and 
Btatlstieal analysis. . 



Monte Penney discussed the application of a research manage nient 
procedure, the Convergence Technique developed by I.ouls Carrese and 
Carl C, Baker, with the Heading Research Program sponsored by the 
U. 3. Office of Education. Fenney candidly pointed out some of the prob - 
lems that were facsd In applying the technique and, in addition, discussed 
implementing procedures for the technique. 

J. William Smith and John L. Yeager discussed the steps necessary 
for planning and implementing a research and development program. The 
phases of R&D planning that they see as critical to the process include: 
1) policy review, 2) strategic planning, 3) factual planning, 4) allocation of 
resources, and S) monitoring. In addition to descriptions of each phase in 
detail, they provided flow charts Identifying the component of each phase. 
Smith and Yeager also included a model for the, evaluation of management 
planning techuiques, and presented illustrattve data which compared 
tschnlquos which aTe commonly used In the pianmng process. The 
communications techniques which were compared included bar-charts, 
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PERT, Convergence, and PPBO. 



In the next seotloQ of this report, the papers read aM discussed 
at the SIG aymposium are presented. Addlttonal writtefl Information 
or aie pro duets associated with each system may be obtained hy wltini 
the autfiors, ' 



A PROTOTYPIC KEOEAaCH iVIANAGEMENT INFOBMATION SYSTEM 

Don B. Croftl 



1. rNTBODUCTION 

Following the passage of the Cooperative Hesearch Act of 1954 th^ 
number of R&D projects conducted by educational institutions has incTGaaed 
substantially. The massive financing of E&D projects by the Fed^^rai 
Government has had an impact npon the procedures for managing and ao - 
counting for the funds aseoclateu with these projects. Due to the number 
of M&D projacts now being oonduoted. it is evident that there Is a need to 
devolop new systems for organizing and managing the information associ- 
ated with the iim activities conducted in an organization. 

The purpose of this paper is to dasorlbe a prototvplo system for 
managing the information about the R&D aotlvitiea conducted by an organ- 
ization. The system outlined in this paper was intended for iDdividuals 
performmg research u^anagemeQt functions within an orgaolzatloa. As 
the system has befln ooncelved, the concept of research management 
includes three general functions. The toree research management func - 
tions iQcIude: 

1. coliectlng descriptive Information associated with the B&D activ- 
ities in the organizations 

2. conducting objective and systematic analyses of the information as 
a basis for reporting the accomplishments of the program 

3. providing evaluative reports about the organization's R&D projects 
for input to the decision-making process 

At this point, It is Qecessary to define what is meant by the three 
terms description, analysis and svaluation. 

Descript ^- refers to reliable and valid Information about the py ojfflctfl 
being conducted, and the personnel associated with the organization 

refers to the formulation of statistical or analytloal reports 
about R&D projects and organizational activities. Evaluatlon- ^refars to 
obtaining ratings from qualified outside oonsultants about the organization 
and the projects conducted by staff members. 

Once the position has been taken to develop a system for collecting 
information about the E &D activities being performed by the organization 
then the next step is to select the kinds of mformation that should be 
included in the system. As one looks at an organization conducting E&D, 

Don B Croft Is Director. Claude C. Dove Learning Center, New Mexico 
utate University, Las Cruces, New Mexico 88001 
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he Is overwhelmed by the complejdty and multiplicity of the operations 
occuring in the orgaalEatioa, At tlils point, it is easy to state that It is 
impossible to develop a system which aompletely describes all the activi- 
ties that occur. In all probability this is correct, but on the other hand, 
rescarah managers must b^gin, and they begin by choosiag what Informa- 
tion should be coilented, and devoloping. ''working systemc^' for Grganiziori 
the information. 

When the activities cf an R&D organization are examiMd cioGOly, it is 
oxsoovered that the following three terms aye useful categories: 
1» Personnel 

2. Gubstantive Program 

3 . Fiuaneial 

In other word^, the activities of an H&D organlEation may be as^bltrarlly 
but usefully, elasGiaed into three components: people, projects, and money. 
At this point, it la impDrtant to dellmtt the scope of the Prototypls Resoarch 
Management Sj^^tem. In this paper only the system for organizing and 
managing Information about the personnel and substantive projectB catsgnr;es 
is diocussed. No mention of the operations, planning, Rud flnanoial aspecrs 
of research managament is presected. 

However, bsfore the research management system is discussed^ It is 
important to present some reasons, why an organization should use a systeni 
for managvng research aofclvitles. The following list provides some of the 
reasons* ' 

L to Inform others about the pr^oiects being conduoted bv the orran^'za-- 
tion ^ . 

2. to provido Informatlcn about the projacts to individuals in the orgau=- 
IzaiLlon who manage the projects 

3. to meet tlie ft^oquent inquiries from outside individuais about the 
R&D aotlvlWes cf tJiQ organization 

4. to provide a basis for condueilng analyses and evaluation of the R&D 
programs 

5. to assist in meeting the requirements for reporting Information to 
agencies which financially support the projects being conducted 

6. providJ.ng procedures for teaching new staff members about the 
organization 

7. providing evaluative feedback to uismbors of the crgai ligation 

A rusearch management system Is composed of separate Gomponents, 
each dascrlblng salient facets of organizational activity. The following 
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outline describes the type of information includsd in the Prnto^pic Pesearch 
Maaagement System* 

L ongoing projects 

2* oompleted prajects 

3* completed prcducty and materials 

4, staff members 

5* ccnsultaats 

6, organizational progress reports 

7, preppring proposals 

8, evHlu^tion reports 

III the neKt section of this report, each component of the system Is 
dlsoussed, 

Ih CUBEENT PKCJEGT3 

E &D organi^.ation8 receive requests for information about the projects 
that currently are being conducted by staff memberi. Meeting these re - 
quests for Information about the substantive projects may be a burdensome 
task unless procuduros are established for retrieving information quickly 
DUd la a form which communicates to others. In addition, It is also 
economienl if the Information has been prepared in a form that can be 
easily reprodiiced by a copy machiue. In this seotlon, the following three 
oomponenta of the £ystem for retrlsv^lng Informatlcn about the projects 
currently imierway are discussed^ 

i. Orgamzaiional rteaume'^ 

2* Project Register 

3* Project Resume^ 

4, Project Supporting Documents 

The resume for tho E&D organization briefly describes the cverall 
purposes of the cr^anieiatlon and delimits the major projects v/Mch are 
beitig conducted. The rSBiima^also includes infcrmaticn about the i^taff cf 
the organizattou. scurceB of funding, aad other relQvai.t information. 
The resume, aJihcngli it is brief, provides an overview of the organization 
and the projects conducted by staff members. Tho organlzattonai resume " 
Is vmtteii to communicate to a wide variety of audiences^ 

B. Project Register 

The pr^ect regigtar is a list of the projects currently being conduct- 
ed by the organizatlm. The project re^-ter also includes a unique num- 
ber for each project and the name of the principal Investigator. It is the 
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key document for identifying the subitantive activities conducted by tlie or- 
ganiEution, and is used as an index for locating information about the pro- 

In many cases It is possible to arrange tho projects listed on the 
project register Into meanlngrul categories. The categories may be pro- 
g-rams, for example: a series of related projects on the same general 
topic or a category describing types of activity, for example: basic re- 
search development and service. Assigning projects to programs or 
general oategories is of assistance In locating speolflo projects. In addi- 
tion., the organization of the projects Into programs assists In defining the 
major programs of the orgaiuzation. & 



C . Project Res limp. '' 



A project refers to Identiflable research, development and service 
acbvities 01 the organizatioii. Tho resume'deacribes what is being done 
how the project Is being conducted, and the terminal objectives of the 
project. The project resume'provldes a brief description of these activi- 
ties and serves as an operational definition for the activities being performed, 
ihe following categories are uaefui to describe projects and additional ones 
may be included if more Information is required. 

1. principal investigator 

2. title of the project, address, date rnltlated 
3„ key word identifiers 

4. purpose 

5. method 

6. ch&raetsrlstics cftiio people 

7. expected terminal objective and date of completion 

D. Prolect Sup porting Documents 

Documents which provide additional Information about the projects are 
called project supporting documents. For example, a project proposal 
suppuos information In more detail than the abstract and is used to supple- 
ment the project abstract. In the same manner, other written materials 
may be prepared to describe the curriculum, film, videotapes, and 
equipment being used. Each one of these descrjlptlve documents provide 
additional Information about the activities being conducted. In some cases 
tue aocuments may simply describe the events which have occurred. How- 
ever, m moat eases the proposal or written plans for the activity are all 
that IS needed to provide supplemeutal Information about the project 
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m. CCMPLETED OUTPUTG 

' organlF,atlons complete many feports about tlie activities which 
Bxe conducted by the organization. The organlEatlon also receives many 
requests for Jnformation about these projects. Accordingly, a system 
similar to the one for retrieving Information about ongoing projnnts Is 
useful when answermg requeata about projeots svhlch have l^een completed. 
I he oomponeuts of m system for organizing Information about completed 
actmties of fne organlzatioii lnclude= 

1. renort reglater 

2. roport abstract 

3, final reports 

4, p;:cdupt file 

The raport register is a list of all the reports which have been com - 
plated oy the organization. The format of the report register is similar 
to a bibliography, it providas the author(s). title, publlaher. and the date 
ot the document. It is a key dooumeDt which describes the reports which 
have been completed by the organization, and Is useful as an index for 
looatlng Information for retrieving publications. 

In some cases It Is neceaeary to arrange the report included In the 
Report Register under different categories for olasslfylng the documents 
For example, the report register may be arranged according to program 
categOTies specified by the project register. TMs arrangement of the 
report suppUes information about the reports produced from each program. 
The report repster may also be organlr.ed according to general categories 
which may assist m looatlng reports requested by other Individuals. 

B. aeDOrt Abfftoa^ 

An abstract of a report Is an effective way of quickly communicating 
the findings of a otudy to others^ E&D organizations reoeive many 
requests for Iriormation about the findings of studies that f'sve b-en 
completed. The abstract provides a connlse and eoonomtcal moans fo- 
disseminating Information to others. In addition, the abstract allows 
research managers to orgaQlzQ and presetit the outputs from specltic 
programs. 

■ The ERIC pnmomm MMSS^mMmpm Hs well as other infor- 
mation dissemmation systems provide examples of ijifonaation dense 
abstracts. The abstraots are short, and contain the essential information 
included in the report. When preparing an abstract the following catego- 
ries are used. Additional categories may be added, if oeeded. but It Is 
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essential that the abstract remain as oonclse as possible. 

1. author(i) 

2, title, publisher, date completed 
3» key-work Identifiera 

4. purpose 

5, mothod 

6, charaoterlstlcs of sample 

7. fladings 

C , Pubj ication Li brar y 

Th'j reports and other publlcatioas of the EStO organlzatloa are 
maititained in a location which permits easy accosa to ooples of the publi- 
cation. The doauments are arranged in a manner similar to a library, and 
it may be neeessary to have a person assigned the responaibility of main - 
tainlng the publication library. The person In charge of the library may set 
up card files, report regioters, and other, cross-reference dooumouts to 
assist in retrieving specific reports, 

D, Product .F ile_ 

One purpose of development projeots is to producG a product which 
can be used by other people. The term product refers to some type of 
materl al objeot which is prDduced as a reault of an B&D activity, A 
product includes for exampjoi currioulum materials, eleotronle equip- 
ment, programmed Instructicjn techniques, videotapes, films, etc. An 
organization may produce many different products, and a system is need- 
ed to have Information about these produpts. Accordingly, the model used 
to organise Information abort the reports produced by staff members was 
adoptsd to handle products. The r9aume''fo- tha product, hov/ever, 
inoludes dlKereiat categories cf informatioQ, These topics are included 
in the product rasums: 1) name of produot dwelorer, 2) name of product, 
3) purpose of the product, 4) cost and availability of the produot, and 
5) tasting and evaluation data abnnt tlis prnduot, VWth the axoeption of 
different catogorios in the r^eumo'the followng onraponsntg degoribe the 
aystem for managliig information about t'lo produoto of an orgarizatlon- 

1, Product ERprtiter 

2, Produot Hesume'' 
3u Product "Library" 

IV. PERSONNEL 

Informa*ion about the background and r.bilitles of the Inrilvidiml staff 
msmbora Is used often in tlie preparation of proposals, nBsJgumeuta In 
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consulting positioQS, and describing the capabilities of the organization. 
Accordingly, it is helpful if reoearch managers have information about 
the staff in a readily available form. This subsystem of the reseai'ch 
managemeat system describes several procedures for organizing the 
information about staff members. The foJlowlng four oomponents of the 
system are deserihed, 

1, staff register 

2, Eta£fvitae 

3, staff publication list 

4, consultant fllo 

A. Staffl^e^tw 

The staff register is a list of the names and positions of the members 
of the organization. It includes all the personnel in the organization, and 
may be arranged according to categories of positions. The catagorles for 
CKample may be professional, techiiloal and clerical staff raemb-srs or any 
other categories which are of assiatance when using the staff register as 
an index for securing additional Information about the staff member. 

B. Staff Vitae 

One of the most frequently used doouraents In a research organization 
IS the vitae of staff mambers. A vltae Is included in proposals prenared bv 
the E&D organization, in documents describing the capabilities of the 
organization, and when Individuals with apaclfic abilities need to be id«n - 
tified. Because ths vitae ia used very often, it is helpful if all vltae are 
m a oommon concise format, and are in a form which is easilv reproduced 
on a copy machine^ A one-page vltae is usually suffloient for most n^eds 
. for information. Aocordlngly, the ExperiGnco, Consultant, and PubUca - 
tioQ eategories arc usually ili'ii^ed to ftve.autviea. Thfj followiuf-- describes 
categories of icformatloa about each person that are included in I3,e v<ta- 
1. Name 

3. Position 

3., Addrfisa and telephone number 

4. SpeoiaUies 

5. Education 

6. Exporleiice 

7. CoDoultant Activities 

8. Publications 

C. Statf^Jn bllcatioa last 

Thli liet is a bibliography of all the publications of the staff members 
A separate list is prepared for each staff member. It Is usually prepared' 
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m bibliographic sfyle, e.g., name, title, publisher, and date, and It may 
molude several key-work Ideatifiers about each pubUcation. The key-word 
Identiflers Idantlfy the nature of the publication, and helpful %vhen 
soreeniQg the blbUo^aphy of the individual. In order to sel-jct appropriate 
publicatioiig to inoluda on tha vitaa, 

D. fisSsuitantFijQ 

Consultantfl art employ-d tc assist Btaff m .mbers In planning, raviov/- 
lag , and evaluating ths projjcts that are conducted by the organlaation. 
Ths rota of an outaido consultaLit Is to prm'ida objaotlve and unbiaoed itsfor- 
mation to tlie members of tiis organization on tcploa relevant to his experi- 
ence and background. Accordingly, anothar component of a research msnagS" 
meat system Is to provide informatioa which assists the members of the 
organization to select qualified outside consultants. 

The system deserlbsd for organiziiig informatioa about ths staff mem - 
bars of the R&D organizatica serves as the model for developing the 
CoosultaQt File. One exoeptlon oocurs, however. Instead of maliitaining 
a file of pubJlcatlons for the peraon, it is neceaaary to maintain a file of 
the reports made by the Consultant. I'he roports may lacluda the evaluation 
cf specifio projects, reports, progTams or the entire organization. The 
foUowiiig comprlsea the Consultant File- 

1. Consultant i^egister-Name, address, telephone, etc. 

2. Consultant Vitao-Speclallies, education, experience, etc. 

3. Consultant Report File-Evaluative reports made by the eonaultant 

V, PEOP?SAL P«EPAHATICN PI^CCEDUBES 

Staff memhsrs in E&D organizations prepare many proDosals for pro - 
jects. ThQ proposals for H&B aot'.vitles are submitted to agenoien for fund- 
ing or to the dlTeotojr of the orfanlzatioQ im oonalderntlon as a potential 
new aotivity. Many new staff members may have never prepared a propo- 
sal, and they may not know the IrBtructlon.-s and prooediires for submjtting 
them to a f'.md.iag agenoy. Accordingly, another important compcnont of a 
rQsearoh management system Is to provide Information for preparing 
proposals to ihe staff members of the organization. The following are the 
essential ooinpononts of this system? 

1. Funding Agoncy File 

2. *'tcbas-for -Projeois" File 

3. Proposal File 



The forms, instructiOQ| brochures, and curreat priorities for projects 
for appropriate funding agencies are maintained In this file. The file per- 
mits staff mombers to gecirre Information wMch assists them In preparing 
a proposal for a specific agency. Maintaining a file or relevant information 
about the submission of a proposal to an agency obviates the dilemma of not 
having the forms, etc., when notice of a request for proposal is given* The 
file is indexed by agency. Additional indexes may be prepared to list agen - 
cies which fund specific types of projects. The Hie may also be used to 
maintain other publications related sourgee of funding for eKampie^ the 
publication Business and Dallv Commerne. 

B. ^'Ideaa-for-Proieot s" File 

The 'Tdeas-for-ProJects" File is used to retain brief descriptions of 
Ideas which may be developed into E&D projects. When a staff member has 
an idea, a brief resume''is prepared so that the essential elements are word- 
ed. One page is usually sufficient to record a title, a description of the idea t 
and references which provide background information for the idea. 

This file is especially useful when there are staff members available to 
develop the idea* For example^ graduate assistants may be used to complete 
the groundwork needed to formulate a proposal from the brief one-page de - 
soription, Thus^ the file is a source of Ideas for individuals who wish to 
Initiate research and development projects. 

C. ^posal File 

The proposal file contains copies of the proposals submitted by the R&D 
organlEation as well as other proposals which may be of assistance to the 
staff members* The preparation of a research proposal is a time-consum- 
ing taskj and represents the Ideas and worh of many individuals. The pro- 
posalj even thou^ it may not have been funded, may contain valixable infor- 
mation which may assist staff members of the organization in preparing 
new propcsals. Some por^ous of proposals remain relatively constantj 
for example the description of the computer facilities available to the organ- 
ization, or reviews of the llterautre on specific topics = This Information with 
ravlsioris according to events which Imve oceured may be of assistance in 
preparing a new document. 

Another value of a prcposal file is that it provides exampies of b^.- to 
write a research proposal to new members of the organlEatlon. Thus, tho 
proposal file may assist new people in improving their skills at writing 
research proposals. In addition, the ideas contained in the proposal may 
provide the Impetus for the development of other projects. 
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VL ORGAMZATIONAL I^EPCRTS 

R&D staff members prepare reports for describing the activities, 
accompliahments, and plans cf the organization. Typically, the reports 
are prepared to describe the events occuring In the organiEation for specified 
pGrlods of time. Annual reports, for example, are pre pared to oommunlcate 
whRt happened during the prior year, as quartsrly reports on the other hand 
may only cover events during a three^month period of time. In some organ- 
izations it is necessary to prepare a report which describes the plans for the 
future. This report may project planned activities for the next year or for 
a longer period of time for eKample, a five-year period. In short, organl-^ 
zational reports describe the past, relatively recent events, and future plans. 
The foUomng are the components of an organiEatlonal reporting system, 

U Annual Report 

2. Progress Reports 

3* Program Plans 

A. A nnual Rep ort 

The annual report is the basic document for describing the activities 
and accomplishments of the organization for the year. The purpose of 
the report is not only to describe the events of the year, but also to provide 
the reader with some information about the history and mission of the organ- 
ization. In other words, the annual report- sets the yearly activities In the 
contejrt of the goals cf tiie organization. The annual report is a document 
which by and large is sumcient to provide an individual with basic knowledge 
about the organization. Annual reports are often disseminated widely. 
Accordingly It Is necessary to prepare the docu_meQt so that it is written 
using terms that will communicate to the audience that rr-ads it. The 
follomug is an outline of topius Included in a typical annual report, 

1. Staff members of the organization 

2. List of projeyts currently being conducted 

3. Brief hlstovy of the organization 

4. Overview of orgauizational goals 

5* Resume of major programs or projects 

6* Activities and accomplishments for the year 

7, Revlow of major acoompllshments 

8*. Overview of future plans 

S« Pf JA^fiQJiorts 

A pr ogress report provides information about the events which have 
rscently oocurred Ir. tho organization. Progress reports are published 
Quactorly or seml-*annually and are used primarily to update the iDiorma- 
tion contained in other reports. The progress report also allows the 
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organization to report events of sigmflcance more rapidly than If it must 
wait until the annual report is %vritteD. In short, progress reports provide 
a means for researcii maQagers to have current information about organi- 
zation acttvitlea. 

One purpose of the progress report is to update the information base 
whlcii is maiQtained for tlie organization, Acoordlngly, to nilnlmlze the 
amount of Information which has already beea submitted, a system of re- 
porting by exception Is preferable. When the reporting by exception system 
is used, the report is prepared by Including new Information when it is 
needed and reporting "no change" in categories which have remained the 
same. The progress report is then used to update the basic file on the 
organizatton. The following Is a list of categories included in a progress 
report. 

1. Overview of organizatinnal goals-update includes new overv:iew 

2. Staff Register-update includes vltae for new statt members 

3. Organjzational ohart-update 

4. Project Begister-update includes resumes for new projects 

5. Report Reglater update-lncludss abstracts of sompieted reports 

6. Disseminatian activities 

a, workshops, coaferennes, etc., sponsored 

b, newepgper releases, lectures, reports, etc. 

The p:cogr am plan deserlbes the long-term objectives of the H&D 
orgauiEation, and ihe nev/ projeots whieh will he Imtlated by staff mem- 
bers „ The program plan lucxudss detailed proposals for the projects which 
have been selected to become a part of the ongo,*ng activities of the organi" 
satio'i. 

The Prcgram Plan is wltton after "IcUonse" seeslons have Men. con - 
diicted to review all proposalE for new projectG, Th^i program pt«n t'\en 
servea as the basis for guiding the plani a£ th9 organizatirn during tlie 
nest period of time. In addJtlon, the prog7'n.ra plan also provides laput to 
"Inhouse" sessionp, for planning the long-term objectivos of the oreaol- 
zation. ■' 

VII. EVALUATION HEPOHTS 

Another componenl; of the PrctotypAc BeHsaroh Management System 
refors to evaluation j.-eports describing the E&ls orgcaiiatiou and the 
projects that are coiidiJctad by staff memhera. Although few syBlematlc 
and cbjeotlve methods have been developed to evaluate organizations and 
projects, evaluaslons are conctantly being conduoted. In this section, the 
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system for organizing information is described, and a form for evaluating 
research reports is preseoted. 



A. EaESEtiSlijiatlgnForm 

The purpose of the H sport li valuation Form is to obtalc ratings from 
qualified outslds consultants about the "qunli^" of a resaarch report. The 
form was oonstructsd primarily as a metns of securing descriptive infor- 
mation about specifto categories which are commonly related to evaluatton. 
The cctegof lea are general eaough to be used for rating d?iferent types of 
projects. Gephart also has construcf:ed an excellant evaluation form which 
thoroughly covers the design and atatistioal aspects of research proJeetsS 
as %vell as other oateEoriGs of information relevant to evaluation. 

Ihe Eeport Evaluatton Form was administered to 53 persons who 
oompleted the form on a research report. The evalimtors had training in 
the topic of the study that they evaluatad. A factor aaraysis of the itims in 
the Beport Evaluatton Form was conduoted to determine If the ItemB obtained 
high loadings on the ccnstructs measured by the Instrument. The rrr,ults of 
Table I indicate that Urn items in each category do fall out cn the same 
factors. Acccif.'iicgly , it v/as conoluded that the instrument does measure 
the foilovvlng attributes of a research report. 

L 31gnlflcano3 to Education 

2. Coneeptuallzatlon of the Problem 

3. Methodology and Design of the Study 

4, Interpretation of the Results 

5, Usefulness of tlie Beport 

Subtest scores for eaoh of these eategorios may be obtaified by oomputlijg 
factor scores for the Instrument or summing the responses to each Item m 
the Eabtest aftgr reverse scoring the appropriate items. A copy of the Repon 
Evaluation Form Is ijiohided in *hfi AppendlK. 

Ival^atlqnMif, 

B&D organt3atioQS reoelve oite visits for evaluatlag ths projects and 
operations of tie ovgauis'.atlon. The sits vioitors usually prepare s'. written 
document v/hich reports their observftticns about the vlKlt. The ^oport that 
is prspared provides feedback to the staff members of the organization, and 
prOT ldes additiona!. input to the deciolon-makiDg prooeos. Thg Evaluatl'on 
File allov/s research managers to maintatn the reports written by site visl - 
tors, an d use theas reports to prepare summaries of the findln^g of the 

"Gephart, william J,, "Kaveiopment of an icstcument for Evoluatlag 
Educational HeaeaToh Heports," Coop",rn.in.vo Research Project No. S- 014, 
University of Wiscousln - Mllwaul:eo, MiLwauIcee, V/iscocala, 1004 
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TABEE I 

VAHIMAX HOTATED FACTOB LOADINGS FOE THE 
RESEARCH REPORT EVALUATION FORM 
(N^S3) 



Item No. 



n 



Factop Loadings 
IV V h2 



Item 



1 


01 


84 


24 


00 


13 


78 


2 


15 


84 


-03 


32 


05 


84 


3 


06 


89 


-03 


17 


03 


83 


4 


07 


85 


19 


-20 


23 


56 


5 


00 


81 


33 


02 


11 


78 


6 


11 


35 


19 


25 


58 


57 


7 


-01 


04 


16 


06 


59 


38 


8 


-02 


18 


33 


31 


65 


67 


9 


29 


03 


12 


00 


73 


64 


10 


42 


11 


-04 


-07 


75 


76 


11 


38 


38 


-18 


23 


46 


59 


12 


81 


14 


-01 


-06 


23 


74 


13 


83 


00 


-04 


03 


16 


72 


14 


72 


06 


33 


-41 


11 


83 


.15 


83 


13 


02 


"03 


24 


76 


16 


75 


-02 


02 


29 


OS 


66 


17 


76 


-04 


24 


-22 


21 


73 


18 


70 


13 


22 


31 


06 


66 


19 


84 


05 


17 


26 


-04 


81 


20 


53 


20 


58 


03 


05 


67 


21 


-01 


21 


76 


20 


08 


64 


22 


28 


16 


76 


12 


28 


81 


23 


41 


-02 


61 


30 


42 


59 


24 


-03 


-39 


63 


17 


35 


78 


.25 


26 


02 


24 


68 


-08 


81 


26 


29 


11 


24 


21 


47 


71 


27 


34 


62 


10 


55 


21 


42 


28 


-XO 


16 


02 


66 


27 


69 


29 


-01 


01 


26 


63 


25 


53 



II Si^ifiaarica 
an important problem 
additional study needed 
hl^ priority topic 
relevant to broad segment 
high potential value 

V Conce ptualiEatloti 
eetabUshed a preblem 
based upon prior research 
related to theory 
objeQtivas identlfled clearly 
variables defined operationally 
variables relevant to problem 

I M a thsd & Design 
controls for coafounding 
statistics appropriate 
ean replicate study 
rellabla and vaUd data 
confounding effects Identified 
assumption not vlQlated 
inferences warranted 
inferences & opinion separated 

IE interQ^tatlon 
findings stated clearly 
closed a gap In knov/ladge 
objectives ^of study fulfilled 
most important resulta identtfJed 
important contribution 

ly __TJg e f uln e s s 
interpretetlons are referenced 
describes use of findings 
Impetus for addltlonEl studies 
Identified new variables 
elarlfled prior flndlngs 



9.1 
33 



4,2 
47 



3,4 

55 



1,9 

62 



68 
* 16 - 



Eigenvalue 
% Variance 



site visitors* Although the e\^aliiative reports prepared by site visitors do 
not usually follow a common format, toe following catagories of information 
may be rated by site visitors. 

1# SIgnlficanee of the research topics 

2. 3opMsticatlon of research design and methodolo^ 

3. Sophistication of development acttvltles 

4. Effacttveness of dissemination aotlvlties 

S* Balance and inte^ation of toe H&D actlvittes 

6. Economic efflcieney of the orgaMzatlon 

7. Effecttveness cf tostitutional Beiatlonshlps 

8. Quality of Personnel 

9# Program Planning and Evaluation ProGedures 
10, Management of the Organisation 

OVEHVIEW 

The research management system which has besa desaribed in this 
paper was designed primarily as a manually operated system* The reason 
for this is, of coursep that many R&D organJ.Eatfons do not have computer 
systems wbleh can be used to perform many of tte retrieval and updating 
operations. Nevertheless ^ the system which has been descrihed may be 
converted to a computerized system* The conversion of a manual system 
to a computerized one is muoli easier and less expensive when the Infor-' 
mation has been rocorded in a common format. This system, puts the 
information in a common fOTmat. 

One procedure whieh was not discussed in the paper Is the main - 
tenance of an updated looseleaf notebook containing the information used 
most often by research managers* In the Gotebook ttte latest abstoacts, 
resumcsi and -vlias of the projects and personnel are maintained^ In 
additlonj the registers for tliese documeutpg are also Included In the note- 
book. The Information supplied in the progress report Is used to update 
tlie notebook, and thus current descriptive Information about the H&D 
organization is at hand. 

In summary, the Prototyplc Research Management System provides one 
way of managing information about E&D organiEationSp Admittedly, it is 
not a final system, but at least it is a start towards Improved research 
management* 

One final note, for the researcher his terminal product Is usually a 
research report of some type which serves to oommunlcate his work, 
flndingSi and conclusions along vdth other descriptive information. The 
utility of his work Is dependeT^t on many factors and the follpwing Besearoh 
Heport Evaluation Jorm oflers some questions which can assist both re - 
searchers in the doijgn and evaluation of their projects as well as consumers 
of tiie research to relate the product to their needs in realistic ways* 
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Besearch Report Evaluatloa Form 
Don B* Croft 
New MeKico State Unl versify 

This form Is used for evaluating rese^.rch reports and was designed to 
obtaia rating and written comments about seleeted attributes of a final 
reports The attributes that have been selootad Include some of the 
criteria mmd for evaluating reeearch projects. The completed Besearch 
Report Evaluation Form pro^des informatlcn from educatton specialists 
for evaluating the quality of a report and is not mm only source of evalua- 
tlve Information about a projeet. Please consider your comments and 
rating oarefully In ovdw to pro^dde accurate and useful information a - 
bout your evaluation of toe report. 

The farm is used to rate mEny different types of final reports. When 
an item not relevant for a specific type of study, please circle MR, 
\Vhen y^u are not able to maice a rating based upon your knowledge 
about a speclflo item, please circle Nl, However, If at all possSjle 
please attempt to aaswer every Item, making careful estimates when 
It is necessary* Please v/rite your comments In a brief and conslse 
manner covering the salient points wMch you wish to make. You may 
Include references to other studies In order to provide us \^'th 
additional information^ 

Please complete the following^ 

Title of Eeport^ 



Author of Report 



Evalnator 



Date 
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hm - Not relevant 1. Strongly dlsagi^e© 

DK - Doa^t Icnow 2, Disagree 

NI - No informatioQ 3, Agree 

4. Strongly agree 



A, SigDlflcance to education 



!• The topio of study is relevant to an Irnportant 

eduoation problem. i 2 3 4 NH DK M 



2, A need eKlsted for additional study on this 
topic. 



1 2 3 4 ITO DiC m 



3. A high priority currently eKists for studies 

on this topic. I 2 3 4 NI? DX KI 

4g The topic has relevazice to a broad segment 

ofeduGators, 1 2 3 4 NR DiC NI 

5^ ringlogs on tMs topic have high potential value 

to educaticnal knowledge or practice, 1 2 3 4 NH DIC NI 

Please describe the value and significance of tbls tcpio of study to 
eduoattonal knowledge or practlue. 
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NS-i^ot reievaat. 1. atrongly diaagree 

d;c-Do *'t kuow 2. Disagree 

ill - :o i;.formatiori 3. Agree 

4, Strongly agree 

B* Coiicepuialization of the Prcblem 

L Tihe report establishes tha exlsteQce of 

a problem, 1 2 3 4 i;:; m 



2, The study is based upon salient prior 
research studies. 



4, Ihe objectives of the study are ideatified 
and described clearly^ 



6, ' The variables identified are relevant to 
the proMem, 



1 2 3 4 im 



3* ihe relationship of the problem tD theory 

or practice is clearly stated, • 1 2 3 4 NH Da i l 



1 2 3 4 NR D!C M 



6, Ihe variables and terms are defined 

operationaUy, 1 2 3 4 NR D. 



1 2 3 4 NK D i m 



Please describe how well the investigators oonoeptuall^ed the problem 
area and selected variables relevant to the problem. 
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NR-not relevant 
D;C^Don't Know 
NI ^No Information 



Methodology and Design of the Study 

U Ihe design of the study controls for important 
oonfounding effects, 

2,. The statiBtlcal analysis was appropriate for 
the data* 

3* The report provides Informatioa for replioating 
the study. 

4, The data were collected fay reliabie and valid 
procedures* 

5, Effects which may have Gonfounded the results 
are identified. 

6, Assumptions underlying the statistical methods 
were not violated, 

7* The InferenQes made from the data analyses 
are warranted, 

8. Inferences from the data and the opinions of 
the Investigator are separated clearly. 



1* Strongly diaagee 
2, Disagree 



3 , Agree 



4, Strongly agree 



1 2 3 4 NR D^< NI 



1 2 3 4 NR DK NI 



1 2 3 4 NR M 



1 2 3 4 NH Di< NI 



1 2 3 4 NR Die NI 



1 2 3 4 NR D 'i NI 



1 2 3 4 NR DK NI 



1 2 3 4 NB Die NI 



Please describe the appropriateness of the design or methods used In 
this study. 
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hm - Not relevant 1. StroDgly dlsb^ee 

DK - Doa«t kncy 2, Disagree 

NI " No i^ormatlon 3. Agree 

4. StroDgly ap'ae 

Interpretation of results 

U The flndings of the study are stated clearly, 1 2 3 4 NH DK NI 

2. The objectives of the smdy ware ftiimi^^^ i 2 3 4 nr DK NI 

3- The most Importaat results wertf idonttfiGd and 

interpreted. 1 r. « ^ * 

* 1 2 3 4 NR DK NX 

4. The report is an important eoatrlbution to 

knowladge m the flald. 1 3 3 4 NR DS NI 

5s The report helpo elose an important gap in 

ctLrrsiit knowledge* i o ^ ™ 

^ " 1 2 3 4 NR DK NI 

Pleaie deBoribe how well the investf gator iQterpreted the results of the 
analysis. 
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-Not relevant U Strongly disagree 
DK Don't know 2. Disagree 

NI *No informajlon 3^ Agree 

4. Strongly agree 



ii. Usefulness of the report 



1. iGterpretatloni are supported by refer© Gcea to 
othar studies^ 

2» The report degorlbes the uaefulnesa of the 
findlDgSs 

3. The report provides the impetus for additional 
studleo * 

4. The report ideoiifies new variables relevant to 
the prohlam* 

5. The report clariaes the findings of prior 
studlee* 



1 2 3 4 NR DiC NI 
1 2 3 4 NB Die m. 
I 2 3 4 HR DK NJ 
1 2 3 4 NR DK NI 
1 2 3 4 m DK NI 



Please dGScrib© Mow mM this report is to educational knowled™ or 
practice* 



AN OPERATIONS i\i\rfUAL POP R£3Z..r 2 CRCAmZATlOm " 

Baa G, vVoolpert^ 

The Wisconsin fiesearch and Dovelopment Center for Cognitive Loarnlng 
IS a somew.mt uraquo organization at tlie University of Wisconsin. Id many 
ways It s^,ms to sit astraddle the aoademlc/buslness divide wl-h a foot In 
each caicp. The Center is a part of the Univorelty of Wlsconflln. Its staff 
^^H°' f °f Wisconsin. Thoy enjoy and expoct th^ 

worUng oc^Qditions and generally unfettered environment thot typical of 
fccu^ atatu.„ on tl^ oti.er hand the level of aocountablll^. required by the 
Center's haa:.y use of federal funds requires operational procedures similar 
to Uose commonly employed by Independant profit and non- profit Institu - 

It was out of this contar^t that the Center elected to pull togethf-r a 
aum^^ of Offlne Education, Otata. University and Center pcUolos ard 
procedures into a Hiadboolc^ The most renent version Is the fourth - 
vmwn It was current tcv lim 1969-70 academic year. While much of the 
contant is relei^anf; only to the Wiooonsic Centsr, other slmjJar orcpm-a - 
uons may bs able to utilise the Hmjdteokas a model for operations manuals 

fh^ ^ ^""^ Inclusive rather than exclusive 

tm Hanriogp^ was put tcgether. it Is a refercnoe volume. The Center 
3,.^ m noteKpacted to read tlie Hanfeok, cover to cove^ and be familiar 
juui all ox tho material presented. They aroeKyscted to have one available 
and to be fasniliRr with its orgaodaatlon m that they cnii Mentily rrlevant 

3naterials whon needed. It is a piroblem soivl^B resc^rce M^iuB .e^^a 

a onmplate and accurate Index is absoluteiy crittcM. ■ " 

The Handbook it now stacds includes everything fi-om the Cfffce of 
^^catwa x^atioaale for estahlishfng Ceoter^s to how to^u.e the ^^toW^ 
nn^xori on tlje telephone. Beth seem appropriate and useful Inclusions, 
o^rostpro and committee lists however are questlonaH^ inclusions. 

co&s.We lu mem^bogc. than on separate, sasiiy mii,placed sheets: The 

w.«r!f ^' Wcolpert is Director, Operations and Business. 
^^om.m RBm^oh and Development Center ior Ccgmtive Learning. 
TJmversity of Wisconeini, Madison, wiSQonsia . 
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material IncZiided In the fourtli version by chapter la: 



I. The CCL^ IT3 NATIONAL CONTEXT AND 1.EGAL COMMITMENTG- 
B elates the Ceiitef to the larger framework of national educational 
research and developmeat and indicates some of the legal constraints 
of federal contracting. 

11. THE CENTEH'3 GOVEENING AND ADVBOnY GROUPS - Rslptes 
the Center to Federal, State and University inatJtutiona and des - 
cribes the internal governanoe of ths Center. 

III. PERSOmnh P-LICmS and POSITIONG - Describes the conditions 
cf employ:nent for tlie various levels of Center Staff. 

IV. PROGRAM PLANNING AND BUDGETING - Describes the orofrnm 
planning/budgeting process used by the Center. ' " 

V. RESEARaH AND DEVELOPMENT PROGRAM 0UM?,1ABY - Vcrv 
briefly describes the programmatic otructure and thrust of the 
Center. 

VI. THE TECHNICAL DEVELOPMENT PHOGEAM - Desorlbes the 

functions of tlie Center's Teyhnioal Section and indloateo procedures 
andooimtrnli-.ts cor.cerrlng.the ubg of .its reoourcos. Describes in 
detail how to relate to cooperating schools, 

VII. THE DISSEMINATION PROGRAM - Describes the functions of the 
Center's Dissemination Section and Indicates procedures and 
constraints concerning the use of its resources. Describes in 
detail Center documents and their use. 

Vni. OTHER DISSEMmATlON ACTIVITIES GUIDELINES - Describes 
m detail how to acquire the resources neaded to carry out project 
activities. ^ gcrlbes admialBtrative services. 

Committees and Staff - Lists all persons related to the Center as of the 
date oi publication. 

Forms and Outlines - Provides samples of the procedural forma needed 
to function effloiently witlun the Center structure. 

Each new effiployee la given a copy of the Hanribod: at the time of 



employment. About four ttmes each year representatives of cidh nf tho 
Center's service sections meet with new employees to hlghUgJit Important 
procedural statements and other aituatloas that are traditionally trouble - 
flome. The most common use of the tfeadbiMk followlag orleatntlon Is by 
staff to Idsntlfy prooedures. The IMndbeok is also obvlouely useful as 
a control device aud is not Infrequently used to settle disputes. In It, policy 
statements are rlocumented and easily aoeeaslblo. The moat useful ele - 
ments seem to bo taa "how to " statements of Center end Uulverslty 
procedures. 



The detailed p\'ocodurea and forma included in operational handbooks 
malm visible a degree of bureauoracy that Is unpalatable to many in the 
University environment. The 'vViseouflin Center has found, howover, that 
to the extent that a handbook can simplify p.nd faclHtiate an Individual's 
work It v/ill he v/ell received. 

The following pages are Illustrative of the scope and rjontent of tlie 
^ndfessis and mclude a oopy oF its "Table of Contents" and from Its 
"Appendix" the "List of Forms and OutliuoB", the "Prospectus for A 
Reaaaroh Report", and the "Prospectua for A Eeveiopment Project". 
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202 



ProspectUB for a Rasearch Projeot 

This prospectus Is prepared by the PriQcipal Investigator for the 
initiation of any new project and also for the annual program piannlDg 
and budgeting of each continuing project. The headings are intended to 
be apphcable to all Center projects, whether the major effort is research 
with a lesser development effort or vice versa. The Center Director -nd 
other members of the Management Council will provide relevant information 
and assist In preparation of the Projeot Prospectus so that program plan - 
ning and budaotiiig procedures of the Center arc implemeutcd. " 

I.. Title of Projeot 

II, Name(s) of Principal Investigator(s) 

last also full-time employees or others affiliated with 
the project and their specific reaponsiblitles. 

. in. Statement of general objectives 

These objectives are stated In accordance with the Program 
objectives and in a almilar form, 

IV. Importance, need, or Justification 

Cnly a summary is neoessary for oontinulng proiocts. For new 
projects this section should oontalii a concise dascriptloo of 
the educational situation requiring change or the problem re- 
quiring research. The status of present knowledge and the 
anticipated contribution of the projected reE^aroh should be 
indicated. 

v. Strategy or operational plans 

This part is the nucleus of the prospectus, for both a new 
and a continuing project. It should describe the operations 
that will be performed in pursuiDg a researoh question 
line of inquiry. The sequence of activities, airi the specific 
mterim and termiaal outputs should be stated in terms of Center 
publications. 

VI. Target population or experimentaJ, subjects 

The characteristics of the experimental subjects should he 
described as fully as possible., as well as the ultimate target 
population for which ^he researei> findings mil have value. 
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Vn, Expected end products or results 



A statement of end products or results by quarter for the next 

fiscal year md by yaar for the next five years should be 

ladioated in terms of substance incorporated in Center publications. 

The iDformatloD. would he stated something as foliows: 
FY 70 

Aprll--Technleal Beport: New knowledge related to motivation 

and retention„ 

July— Working Paper; First draft of synthesis of research on 

motivation. 

October— Two Technical Reports: New koowledge related to the 

semantic components of ooncept learnlDg. 
January— Working Papen Pilot studies related to acqnisition 

of llteraoy. 

FY 71 



Five Technical Heportsi Mew knowledge related to the effsets 
of situational variables on concept learning, 

FY 72 

Theoretical Paper: A theory of creattve thinldng . 
FY 73 



Six Technioal Hoports: New knowledge related to learning 
and ooclni oonformi^y, 

FY 74 



Working Paper: A preltminary statement related to the structu"e 
of concept attLiinment abtllttes. 

The abovs types of statemoata might weJi follow a brief Intrbduoto-^-y m^'aKraph 
which generally describes the Une or liuoa of Icquiiy opg Ig pii-auiag in a 
research project, EM rPsuits or pirodugtB ehoi-M slivnv a r-rof^roflsioii of 
activities over time toward a refined theory. Finallv. activities shovld 
properly reflect the fundmg level of the project— the larger the projeot, 
generally, the more results or end produotg would be desorlM. 
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vm. Relation of this project to other projeots 

Helatlonahlps should be stated in terms of the substaDtive area 
the general objective, the strategies, personnel, and substantive 
results or outputs, ^ach principal investigator should bo thoroughly 
famlhar with all other projocts of the same ProgTam and with any 
project of similar concern in other Programs. 

K. Time schedule 

A projected time schedule should be Indicated on the Time Schedu'e 
attached to this form, (Gae Form 204) 

3C. PerEnimcl and budget requiremoata 

Indicate on the attached budget form: 



2. 



The prcportion of the prionipji lavesiigator's tlms whiBh 
wiIIIjo committed to this project. 

The number and types of supporting personnel wbloh will be 
needed (excliiding secretarial and clerical staf^ . 
Activities that may require expenditures aboie J250 for 
suppHeff, ortuipmeut, or services. The staff in the Operations 
and Business Section are available for help In eatiraatins costs. 
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Prospectus for a Devolopment Project 

This prospectus Is prepared by the Prinoipal Investigator for the 
inltiaUon of any new project and also for the program plaQning and budgeting 
of each continuing project. The headings are intended to be applicable 
to all Center projects, whether the major effort is development with a 
lesser roseareh effort or vice versa. The Center Director and other member 
01 the Management Council will provide relevant Information and assist in 
preparation of the Project Prospectus so that prcgrara planning and budtrotlng 
procedures of the Center are Implemented. 

I, Title of project 



II. Name(s) of Principal Investigator(s) 

list also full-time employees or others affiliated with th.: 
project and their specific respoaslbllities for the development 
or the related development-bRsed researoh. 

ni. Statement of general objectives 

Tbese objectives are stated in acoordanoe with tlie Program 
objectives and in a similar form. 

);v. Importance, need, or justification 

An analysis of the present elementary schooi situation is given 
with an indication of the need that the proposed development 
is to alleviate. The need this devolopment aotlvity is trying 
to satlofy is clearly specified. In turn, this gives a first 
approximation of the type of product and the target population 
or populations; e.g. materials for use by students and related 
lastructlonal guides for teachers, 

Deficlencios in the presecit situation should be outlined, and 
the status of the target population upon completion of the 
project should he indicated; 



a. What the Individuals In the target population will be 
doing when, they have achieved the objectives. 

b. Under what conditions they will be doing it, 

c. How well they will be doing it. 



V. Strate^ or oparational plans 

The saqueDaal operatioas that vnll ^ performed in developing, 
testing, and revising, the product(s) are outlines. In general 
these operations include 

a, A statement of to© speciflo objeotlves of tfa© pri3auot(s) in 

terms of pupil behavlprs, teacher behaviors, or both deponding 
on the natiBfe of the product* 

The "real world" oonsteainte In whleh the propoaed produot 
must operate* 

c* A set of speclfloations of produGta, altarnattves perhaps, In 
terms of <1) tj^es, a,g, prlntedp audlovlsuals (2) length 
or amount of materials i (3) use of tte product, e*g, Indepen^ 
dent Btudy^ small group dlseussionsp large group Information 
receiving; (4) amount and type of assistance required for 
praotitioners to make the produot operative* 

dp Ine product development * formative evaluation - revision 
sequanoep 

e. The large-aeale field test by the Center or otter agency and 

the summatlve evaluation, 
f* Procedures f or d issemlnation by Center and Installation and 

further field test by other agency. 

These developmental steps are explicated In ''Guidelines f-r 
Quality Vorlflcatlon of Matarials D.^vslop^d at the OfenLterfor 
Pupils a d l^duoators'U 

VL l arget population 

'ihe characteristics of the target populatlou for the Intended 
i.istructio .al material should ho described as fully as possible. 

VIL Expected end prcducts or results 

A statement of end prcducts or results by quarter for the next 
fiscal y^ar and by year for toe next five years should be indicated 
In terms of substance incorporated in Center publications^ 

The Information would be stated something as follows: 
FY 70 

April— Worldng Paperi A second version of a teacher *a manual 
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for kindergarteii and early primary teachers. 

July— Workiag Paper- a pilot version of Individual diagnostic 
or assessment tests, 

Cctobei— Technical Reports Hesults of pilot testing the second 
version of a teacher's manual for kindergarten 
and early primary grades. 

January— Theoretinal Papart A taxonomy of behavioral objectlv 
for kindergartea and early primary levels. 



71 



Matsrialsj Additional system coraponents iDoludlng video 

, tapes and printed iiiser-Ace materials (reported 
In a Practlctil Paper) for teachers, Idadergarten 
and early primary levels. 



FY 72 



TeohQical Heports Eesults of field tests for materials for 
klndergartea and early primary levels. 

FY 73 

Worldng Papari A first version of materials for later 

primary levels, 
TerhMcal Beport: Pilot test results, 

FY 74 



Practical Papers: The aeocnd version of materials for later 

primary levels. 
TeouMcal Heport; nssuits of fie id tasis. 

The above types of statemsiits might well foUow a brief introductory 
pa.vagraph wMch generally describes the nature of the system under 
development, 2nd resultfj or products should show a nroPTesslon of 
^tiv^:.es over time toward the devfelopraeni of an Instruotlonai system. 
Finally, actavjUas should properly reflect the fuuding level of the projent- 
(he larger the proieot, gensrally, the more results or end pr^ucts 
should be desc:rlbed. 



" 38 - 



VIII, Relation of tiiis projcet to other Ceater projects 



RelatlonshlpB should be stated in torms of the substantive area, 
the general objeotivea, the strategies, perBonnel, and ortputs. 
Each Principal Investigator should be tooroughly familiar %vith 
ail other projects of the same Program and with any project of 
similar concern in each other Pro-am* 

IX, Time schedule 

A projected time schedule should be indicated on the Time 
Schedule attached to this form, (See form 204) 

X. Personnel and budget requirements 
Indicate on the attached budget formi 

1, the proportion of the Principal InveBtlgator's time which 
will be committed to this project 

2, the number and i^pes of supporttng personnel which will be 
needed (eKoluding secretarial and clarlcal staff) 

3, actlvltieo that may require expenditures above ^250 for 
supplies, equipmentp or ser^^lces. The staff in the 
Operations and Business Section are available for help In 
estimating costs. 



MANAGEMENT INFORMATION SYSTEMS FOE THE :i:DUCATIOiTAL ^ liSFA^Ca-^ 

John F. ViQsonhaler and Stuart W. Thomas 

Thomas A. Edlsoa once complained that "Genius Is one per cent 
inspiration and -l-ety=nlne per oent percpiration. " Unf ortunatel-, the 
statement is all too true for the acUvltles of the university soientlst. 
As the modern scientific researcher fills his various roles ~ as mm^or 
experimenter, and administrator— much of his energy is spent on tasks 
which make little use of his sclentiac abilities. The present paper 
concerns the use of computers to shift the scientist's distribution of 
effort from perspiration toward inspiration. 

GeneraUzed Management Information aystoma for Heeearohers . 

Over the past ten years, computer scientists have directed their 
attention to the various isolated activities of the modern research 
scientist. Automated systems have been offered for specialized 
literature searching (Kent, 1966), for statistical data analysis (Cooley' 
and Lohnes, 1962), for data sharing (Bisoo, 1870), and even for data 
collection (Uttal, 1968). What has not been considered is the total 
activities of the researoher, i.e. , the need for generalized systems to 
support the combined activities of the modern scientiDt. 

The key concept is the Management Information system (MIS) pre- 
sently used In administration (Weiss, 1970), "An MIS may be defined 
as a system that makes any information in the data base (files) immediate- 
ly available to the user, to satisfy his planned, as well as unplanned 
information requirements." 

II. MANAGEMENT INFORMATION OYCTEMS 

There are two aspects to the MB approach. First, an Integrated 
file IS prepared Including all data necessary for the user's information 
needs. Second, a general purpose file manafement system la used to 
mamt aln flies and generate reports. 



John P. Vlnsonhaler is Associate Professor, Information Systems 
Laboratory, Miohigati State University, East Lansing, Michigan 48823 
and Stuart W. Thomas is Systems Analyst . Information Systems 
Laboratory, Michigan State University. East Lansing, Michigan 48823 
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A. Intefflfated Data Files. 



The oreatlon of separate reports for various maQagemeot purposes 
requires the Integration of many types of InformatloQ, as illustrated In 
the followmg figure. As shown, the MIS user must create a universal 
file of liokod data records. 

The iategrated data file Is capable of two types of reports— those 
gensrated from separate files and those generated by combining separate 
files— e.g. , combining, personnel and budget data. The integration or 
common iinlc permits the preparation of reports for unanticipated Infor - 
mation needs. There are many methods of forming links. The common 
identification orlnhel is the one Illustrated. Records are Integrated by 
selecting those with common label s, A more general method is the use 
of a common index to the contents of all files (IVIeadov/, 1967;pp. 226-2S2) . 



41 - 



J'IGUFE 1, mtegrated Data File, 



Project 
IdentifleatloQ 



Identifying 
information 
used to 
link files 



Personnel 

aie 



ProjSGt I L_ 
file If 



Data for 

personnel 

reports 



Data for 
project analysis 
reports 



^ Budget 
_ fAle 



Data for 
financial 
reports 



- 42 - 



EKLC 



B. File Maaagement^ 



The seaond aspect of tiie MIC approach Is the use of generalized file 
management facilities required to produce both antiGipated and unantlcl - 
pated reports (Aron, 1961), The generalized file management system is 
a set of computer programs which will produce many diaerent types of 
reports depending on the files and report descriptions submitted. Such 
systems are widely a^^llcibla Md are perlodioaXly reviewed in the 
computer science literature (see Olle. 1969; Senko, 1969; and Meadow 
and Meadow, 1970). The use of a generalized file management system 
Increases the flexibility of the MIS, since one need not describe reports 
to be generated before implementing the system. 

The following figure Bhows the complete MI3 concept for research 
projects as typically implemented in a unlversl^ environment. 
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FIGURE 2. The File ManagemeQt System 
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.he university ewironment dictates some special MIS characteristics. 
Very few reaearch projacts iacittde budgets for preparing management" ' 
reports , let alone funds for computers or computer programs Hence the 
research must use locally available facllItlGfl. . The most practical solution 
for most scientists Is to use the university's computing center. Program- 
ming IS necessary since general purpose file management systems are 
available for Qeariy all major computers. (Meadow and Meadow, 1970) 

In summary, this Is the MIS solutloa to the rGeearoh management 
problems — creating a generalized data file from aggregate data records 
an i then ganeratlng the required reports from tte file by usin*' file 
management sattwarm available on tJ^e scientist's own eompute'r. 

Now let us consider a case study of this general technique as appited 
to iinancial record management. 
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in. MISi?: A CASE STUDY 



The present authors have developad a represSDtatwe MIS .for research- 
ers to provide accouatlDg raports for two researoh projects funded by 
grants and contrants from the U.S. Office of EduoatiOQ, Their projects 
accounting proccdurBs related expenditures to reaparch objeciives. 
Hov/everj reconQili.ation of the project aceoimtlng with reports prepared 
by the university bunlness office required reclassification of many 
expcinditures, and furthermore, flnaDolai reports for the USOE diffared 
from both the university reports and the accounting olasfllflcatlon useful 
for the project. Facod with preparisg ceparate reports for the grantor 
and for the university, the authors Implemented an MI3 to maintain inte- 
grated files and generate all reports with the CDC 3500 computer at 
Michigan State University. 

The case study assumes no unusual coraputer capabilities which 
might prove to be uoeA'allabio to tlie average ualverslty ref earcher. The 
computer facilltiea used b^' the authors' MIS are limited to batch process- 
ing with one day turn-around. Further, the file management system usfid 
in the project, the Baeic^Injormatton Hetrieval Syst em (im\3} , is avail - 
able at cost to any univsrslty researcher and may be modified for 
installation on most computers. (BmS is writtsa in standard U5A3I 
FortraQ, with dialects for IBM, CDC, etc. See Vinsonhalfir, Haitersnn, 
and Thomas, 1970.) 

Now let us turn to the case study. To bejln with, confiider tlis 
overall design of MISR in relation to the existing university accounting 
system. 

A. Genera^Desij^ 

The fcllowla,? figure summarizes the overall fitriscture of the MTEE 
used at Michigan State University, A distinction is drawn between the 
unlvorsity accounting and the M!SR operations. 

The MISE at mXJ operates in pai-allel with the university acooiintlng 
procedure (showu on the left in the figuro) , The un.tvQrslt.y rpports arfl ° 
gonerated mainly to veriry tha MISB data records awl to eptabllsh 
accurate imiversity catogorlzations of .all oxpendituves , 
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At mv, costs, appear 1q university records only after funds have 
been legally expended. For this reason, m imB umverslty and gracitof 
reports Include only expended funds. Project reports Include both 
ejcpended and committed funds. 
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j:1GOTE 3. The M13A Case Study at Mlchigaa State University 
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a. Inmt 



Input to the MI3R/MSU syotem consists of fee Intep'ated data record 
wliich combines the basic accounting categories of H3U, tlie USOF, and the 
hUxB Project. A sample data record is Illustrated in the following figure. 
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C. Grantor i^etjorts 



The next figure illustrates the type of report generated by MIQE 
for the grantor, the U.D. Office of Education, bureau of Education for 
the Handicapped, using the grantor categories (as speclfled In the 
<^FC-OE-BR field of tJie data record) to display cumulative totala of 
expenditures, aince the MI3R teohnique allows complete freedom in 
categorization, different reports may be generated for different funding 
aROncies. 



FIGURE 5. C-IANTCjI rEPORT- Expenditures by Grantor Category and 
by Account Charged. 
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C .Projeot Reports 



The following figure shows a type of report frequently requested by 
project admlriistrators. The rsport documeots the manner la which federal 
funds have been expended to meet specific project objectlvei. This Is 
accomplished by having expandlturea distributed botli by project categories 
(as specified In the fVC field) and by the grantor oategoriei (as specified ' 
m the:^FC-OE-BR field) . TMs type of report provides a sort of "monetary 
translator" which indicates how funds are expended both from the point of ' 
view of the grantor and the project director. 



FIGUHE 6. PEOJECT R3POBT3: Total EjEpenditures Distributed by 
Project Categorl(3s and Federal Categories. 
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E. Coat Analysis . 

Although the authors kept careful records, tJie actual costs of MISR 
are only estimates. The problems are that project secretaries work 
periodically on MISB - henoe. personnel costs must be estimated- cost 
parameters such as number of records proeesied vary montliiy; and finallv 
computer time and keypunch rental reflect current hardware oost^ at 
Michigan State University, 

In general, costs for tlie MISE case study appear to be very low 
Uurmg the eight months of development, a total of 525 transactions ' 
were proeessed at a total cost of $1400 Including salary and computer 
time, or about ,2. 67 per record. During the last two months of operation 
a total of 140 transaQtlons were processed at a cost of ,,174, or about 
a. 24 per record, m terms of total project budget, development costs 
were about 1. n and operational costs about 0.0%. This per record cost 
IS about half that cf preparing and mailing a single business letter 



IV. C0WDLUGI0N3 AND DI3CU03ION 



Severai comments may be offered from our experience with the MISR 
technique for research project administration. 

Possible A pplleatlon[3 of the MISR Procedures. 

The particular MK procedure used in our case study could probably 
be applied at any major university by most funded research projscts All 
Uiat would be required is the modiflcation cf tlie MIOR data record to su^t 
the needs of the researcher's university, fuuding agency, and project 
objectives. The rBaearcher could either obtain the required compu*«r 
programs from his local computer or he could obtain a copy of the 
BIH3 system from the authors. 

Further, the MIOH procedures might also bo applied by administrators 
of academic departments and small research institutes. Again, the probl-m 
IB simply to modify the MIGB data record format for the administrator's 
special requiremerits. In short, it would appear that MISR may have many 
useful applications. ^ 
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B. PcMiMG„-Applicrition of the MI3 PrQecdure: 



The MI3 technique is based on two concepts: liitegrated data files 
and generalized computer programs. Intearated data files are required to 
i^crmit the preparation of reports requiring a cqmbmation of several types 
of data. The file integration technique used in the present system Is to 
prepare an Integrated data record which includes fields for common 
information and for special information unique to the needs of those 
sharing the data file. 

General purpose software for file managemant is required in order 
to use local computers and to prepare the diverse types of reports required 
by the MIS users* The MIG procedure is made sconomlcal through the wide 
availability of generalized, machine Independeiit computsr programs for 
information storage and report generation. The MIO technique has many 
potential uses. 

In addition to assisting the research in his administrative and 
scholarly roles, a sufflcieiitly flexible MIS could serve directly in his 
research activities. In the experience of the authors ^ a very useful 
application would require the system to store the researcher*s data as 
he collects it, along with such information as tlie eKperlmentai units or 
subjects i the variables which were measured^ the measui4ng Instruments, 
date of collection and grant numbers, etc. 

One beneflt of such a laoility Is that it would permit the researcher 
to automate data preparation for computer-aided data analysis. He can 
select subsets of data for a given ©xperlment; join data (or subsets) from 
several experiments, do simple toransformatlons and data collapses, etc, , 
witliout having to sort, merge^ or repunch data cards, and without having 
to preprocess data in transformation routines prior to submitting it for 
analysis, rurthermorej some aoftware systems have provision for incor- 
porating data analysis progTams within themselves — as is the case with 
the BIR3 system used by ths authors (Vinsonhaler, Hafterson, Thomas; 
1970: see Volume VII ~ Information Retrieval and Beport Generation), 

Another interesting use of tlm MIS Technique would be In managing 
research documents and materials. Thus, the researcher could use the 
MI3 to maintain files of related projects, and relevant documents and to 
produce needed reports such as bibliographies and lists of refersnces* 
Similarly, the MIS could be used to store stimulus materials and to 
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generate materials required for research. A few researchers have 
reportedly used computers for both documeDt and test Item storage 
(e.g., see 311berman and filep, 1968). The Mio technique might 
provide a practical method of eKteiiding this computer researGh 
assistance. 



In the present paper, we desorlbed a practical MI3 to support the 
scleiitlst in his role as administrator-manager, riopefuny our study will 
stimulate research on information systems which will assist tlie acientlst 
with other major aspects of his professional life. We all know that the 
computer can be a helpful data processor. Perliaps techniques like the 
MI3 approach will now show us practical ways of raaldng the computer our 
all purpose researoh assistant. 
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TOWARDS A TECHNOLOGY OF R&D PLANNING 
Monte Penneyl 

Besearch may be defined as an ordered process for gaimng knowledge. 
The aocretlon of knowledge about a given phenomenon occurs through the 
gradual specification and elaboration of Its properties and its relationships 
to other phenomena, A key point In the development of an area of inquiry is 
reached when the existing knowledge base is siifflcient to permit organized 
attempts at solving problems or adding new capabiiities to the human re 
pertolre. When attempts of this sort succeed, the results are tangible i 
the eradieation of a disease or the completion of a space flight* 

Successful R&D plaiming then, is analogous to scientific research in that 
it is an ItGratlve process for obtaining better and better approximations of 
the logic inherent in natural phenomena. This is not to say that planning 
ls---or will ever be--an experimental science; its role is rather to reduce 
gross uncertaliiMns into smalier ones which y;g^ amenable to scientlflc 
solution. Planning and ressarch, then, caa and must complement each other 
In solution-oriented R&D programs. 

A gap ejdsts, however, between the need .^or B&D planning and our 
present repertoire of formalized te'^hniques for fulfllllng those planning needs. 
At best, we have one teohnlque appropriate for use in Phases I and II of 
Figure I, and two or more for Phases III, IV, and V, Furthermore, the 
techniques for Phases I and 11 were developed In 1970 and 1967, respee - 
tlvely, and have not yet been widely institutionalized. This means that 
the eKisMng techniques must demonstrate considerable success before other 
workers will attempt to develop alternative, and hopefuily, better ones, 



Monte Penney J,s Resoarch Associate, National Center for Educational 
BaD, TJ.3, Offlce of Educations Washington, D,C, 20202 



Figure I 



mAZE'J OF B&D PLANNING 
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Policy ReviGw 


II 


Strategic Planning 


III 


Tactical Planning 


IV 


Allocation of Hesoui*ces 


V 


Monltorlnff 



In order to provida a framework for disQussion, I have pieced together 
a flow chart descriptive of five Phases of R&D planalng. 

Five Phases of Planmng^ Defliied fSggJFlgi^!^^^ 

L Initial Po Jloy Review Phase: 

Gldeonse (Science, Decembar 4, 1970) asserts that the choice of 
educational R&D goals is a valuo-loaded one which demands metlGuloua 
eonsldaration of key policy Issues. Goals survtve or are disji laood, 
depending upon the adequacy of the funding agenoy»s itudies in Ave 
broad areas: 

A. What is the nature of the contexts within which a given educa - . 
tional E&D program operates ? 

(1) Social, poUtleal, economic, and philosophical; 

(2) educational policy 

(3) educational (delivery) systems 

(4) science policy 

B. What is the ultimate purpose (educational goal) of educational 
R&D in a given area? 
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C. What R&D funcfions^ deflnltionSj models, and descriptors are 
basic to understanding of--^and success witMn--a given R&E) area? 

D. Who are the workers (scientists and developers) who will perform 
the R&D and where are they located? 

E. What structures exist for conceiving, planning , and implementing 
R&D programs in a pven area? 
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Figure 2 

.PLANNING AND mPLEMENTATICW OF A RESEARCH AND DEVELOPMENT PROGRA 
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Phase 11- Strategic Pianning 
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Phase III. Tactical Plappinff 
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The initial policy review is complete when program goal has (1) been 
specified and (2) been juBtified from each of tJie five points of view described 

above , 

II. 3ti'B.tBgy Phase; 

Carrese and Baker (Managemeat Science. Aijril. 1S67 > acknowledge 
that the stategic vs tactical dichotomy depends upon one's point 
of reference' top leadership may view a massive E&D progTam as 
but a single tactical element In some larger structure. For our 
purposes, we will define the strategy stage of planning as struc- 
turing cohepent, goal-oriented, multiproject programs. 

A step-by-step prooedure for planning at the strate^ is evolving 
through the operations of the National Cancer Iiistltuta and the U.S. 
Office of Education, Called the Convergence Technique, it will be treated 
hero in some detail because it is the most highly developed of the for - 
malized approaches to strategic R&D planning. 

The developers of the Convergence Technique reasoned that, because 
research is a process of exposing the Inherent order— or logic— of 
natural phenomena, then the logic of the pheQomenon under study is the 
best guide for sequencing and specifying the research needed to get a 
functional understinding of It. Therefore, research conducted according 
to a plan which very closely approximates the logic of the phenomenon 
under jf,udy will progressively reduce the number of unanswered 
quoctf o.-.i, allowing funds and talents to CONVERGE upon the remaining 
que^,... V;;;. TMs is the basic concept of the Convergence Technique. 



*5f'MlIzmc Vi<;.^;un'i,; ,,, of thz technt^ue 



. ;:ral orfCi.Alzmii: " io'i have been found uiJi ": ;:. .i.rji'-ing the 
Convc— ence Tec;^uique q<: . s ^Barch problems. The spurufioafion of a 
final goal is the most critic;;' of these, because it determines the nature 
of the research program. , - rormulating the plan for the Special Virus 
Leii'rumia ProiTi-\m, Nati— 1 ,j:.;Loer InKtit-te planners could hpra chosen 
as r. o;oal to ' ■ alop te-r. ,:,aaj .v,v. Qgi-jy .iiagnosis c<f Jfiiikemia, " Such 
a goal raifelit h;y o requi •■• ' ..m^.'xrzh U - 'Ify the fi-nt syniptomif that 
appeared. Instoad, thej - Oi\ti the goal "rwutine zn,-, w?r'tspread use of 
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preventive and/or GOntrol me^ures for leukemia.'* This goal requires 
Identiflcatlon of the Qauses of le^emla and development of drugs which 
prevent and /or control the disease* An appropriately stated final goal 
should Indicate the Wnds of researeh needed- or as Carrese and Baker 
state, it must ^ 'serve as the basis for action," Goals must neither be 
diffuse (e*g, , to improve manklnd^s eKiBtenee) nor lacking In perspective 
(e.g, , to test some gingle hypothesis,) The Lan^age used for stettng a 
goal should be as concrete as possible; abstraction tends to increase 
ambiguity and often provides elegant though unintentional camouflage for 
empty concepts • 

• Having specified a final goal^ a convergence planning team Identifies 
and sequences the sub-goals that must be achieved to accomplish the final 
goal. The sequence of tliese intermediate goals apprcmmates the logical 
structure of the phenomenon^ given what is known at toe time of planning. 
The establishment of Intermediate goals Is accomplished In the same manner 
as the final goal. 



CRITERIA FOR GOAL ATTAINMENT 



Specification of final and /Intermediate goals provides a skeletal outline 
of the logic system tio be followed. But without criteria for the satisfaction 
of tliese goals it is an incomplete plan , For example ^ if one intermediate 
goal Is to develop a test, then criteria for acceptable validity and^ellabiilty 
of the test and perhaps for ease: of administration and Ecorlng should be 
specified. 

Every research project undertaken in the overall program meets two 
kinds of standards: standards for scientific rigor AND standards for 
relevance to the program^s goals. Because the criteria for goal attainment 
prnvide relevance standards for tiie programp they aid in assessing 
progress, in determining what new Information is needed, and Jn making 
program mtKjiflcatlons. 



THE PLANNING TEAM 

Obviously, the abilities of the people who determine goals and 
criteria have a great deal to do with the plan's ultimate accuracy in 
approximnting tho veal order of tho target phennmenon* Experience 



ladicates that a group of more than five or six will eKperience problems of 
semantic diversity and communication time tliat make planning Inefficient. 
Because of this each team member must represent a distinct, needed 
specialty wth as little ovc ^nn as possible, Accordmg to Carrese and 
Baker, the basic team is coiuposed of three members: 

(a) ^*one person who has a genGral, comprehensive knowledge of the 
broad area of research and the major sclentiflc disciplines 

in wMch the substance of tlie program to be planned Ib Included; 

(b) one person ^th speeialiEed knowledge of and eKperience in the 
conduct of rssearch in the particular area to be planned. . , ; 
and 

(c) one person (usually sufficient) with general system analysis 
knowledge and direct experience In the use of tlie planning 
technique." 

This core membership may be enlarged by adding other specialists as 
needed, up to the practical limit of 5 or 5. 



Tm PLANNING SES3I0NG 



The planning team must be prepared to devote a large, uninterrupted 
block of time to forging the plan. Every idea presented is pursued to one 
of tliree logical end points: 

(1) the concept is relevant to intermedlats or final goals; 

(2) it is not relevant; or • 

(3) with specific modifications, it would become relevant 

This process, which Is both exhaustive and exliaustlng, should 
ideally continue until the logic system is very highly developed. The 
planning team, however, does not work in total Isolation, It should have 
prearranged access to library materials and consultants. Review of 
interim drafts of the logic system can be performed by other experts 
while the planning team moves ahead. . 
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THE COIWERGENCE CHAHT 



The plaoning team obvlouily camiot keep all the relevaQt details la 
mlad throughout a 4- to a*week sessiOKi. Tape recordings and photos 
of blackboard notes are good devlQes for short-^term^storage, but the 
Convergence Chart la the continuously developing model of ttie program 
to be conducted. 



Phase n-1: goal 
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Phase n: goal 
Step 1 Step 2 
Task 1 Ta^k 1 
Task 2 Task 2 
Task 3 Task 3 




FIGURE 3 

Flgura 3 shows the typical format for charting a program logic 
system or linear array. Specified in this array are the activities 
judged to be neceesary to tlie attainment of intermediate and final goals. 

the logic of a given area of research so requires, tiie linear array 
may branch or contain parallel lines of investigation* For example, the 
National Cancer Institute's program on; lung cancer includes work on 
development of a cancer-free cigarette in addition to work on dlagaoBls 
and controls of the disease. The linear array is intended to display the 
best available knowledge and estimates of the work Involved in attaining 
the flnal goal. Because it contains both relatively ^*hard'' Information 
and relatively "soft*^ Information, it Is toe foeus of continuing re-exam- 
ination and modification in Ugit of new flndings. 

Two additional arrays are constructed. The concurrent array 
shows risearoh tasks which provide alternative approaches or promise 
to optimize performance In toe linear array. The supplementary array 
is composed of long-range, ''high-risk'* acttviaes, which, if suecessful, 
could cause major changes in the logic of the linear array. Taslcs in 
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the linear array receive first priority for funding; if all of the tasks In the 
linear array can be funded, than work can begin in the concurrent array. 
The tasks in the supplementary array are funded last. Figure 4 shows a' 
Convergence Chart %vlth some detallliig of all three arrays of the research 
flow. 



The Convergence Chart of a research program also reqiures speclfl - 
cation of a system for transmitting timely information to everyone concerned 
Guah a systsm would include all project personnel, the staff of the funding 
agency, and other interested parties, especially researchers who wish to 
collaborate in the effort and scholars who wish to critique the overall 
plan. In flgiire 4, a minimal infornatlon flow would report program status 
on a quarterly or semi-annual basis, as necessary. The structure provided 
by tlie linear array allows progress reports to be botli oonelse and concrete. 

The Resources Flow section of a Convergence Chart specifies the 
kinds of talents and faciHtlea needed, and the approximate cost of each 
phase of the research. 

Although the Convergence Technique Is a formalized system for 
planning research programs, it is not a closed system. The planning 
begins with the constructiou of a "rough draft" of tlie target phenomenon's 
inherent logic, but this la only a beginning. The plan is not considered a 
blue-print to be foilowed slavishly; instead, it is a structure which is 
expected to receive extensive modlflcatloQ tlirough professional criticism 
and through careful analysis of research findings. 



ni, Taetioal Phaser 

Tactical planning is the designing of coiistltuent project-sized 
pieces of work to satisfy program criteria. Educational R&D 
is stOQgsst at this stage; educational researchers and 
developers are prodigious builders and users of tools for this 
purpose. The tools of tactical planning Include research de- 
sign procedures' conceptual mcdels (eg, CIPP); computer 
programs for costing, controlling, data-analysis, and scheduling: 
and many others. Taetioal planning culminates in declslcns 
about how Individual projects should be aoooxTipliBhed, who is 
likely to have competence in accomplishing them, and the 
like. 
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rv. Allocatloa of Besouroes Phase: 



This is the buslDess of securing eotitraetual relationships with . 
institutions which will perform the work. This stage may occur 
in Universities (through delegation of tasks to staff or graduate 
students); in business or non-profit firms (through assignment of 
tasks to a lab or division); or in the Federal government (through 
assignment to staff scientists or procurement via negotiation or 
RFP). 

V. Monitoring Phase: 

As a research or development project progresses, tliere Is often 
a need to re-deslgn portions of it, and efflcient re-desIgn requires 
constant accurate information about ongoing work. 

Monitoring isdeflned by Terselic and Carrese (National Contraot 
Management Journal, 1989 V.3 No. 2) as "The function or'p^cSIsTf 
continuous evalufttioii of an ongoing research and development task by 
scientists expert in the subject matter of the task, in terms of 
achievement towards task objectives and, with concern for resource 
utihzatioa. M Monitoring then, la "an evaluation function requiring 
a gathering of information depicting progress." 

Viewed as simple system, project monitoring has these components 
and proeesses, as shov/n in Figure 5, 
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, FIGURE 5 
PROJECT MONITORING 



Process 



Outputs 



1, Design of a given 
R&D task. 

2. Management plan 
derived from (1) , 



1- Analysis of Input 4 



(a) quality of work 



1. Report deslgiie 
for 

(a) project officer 



3. Reporting plan 
derived from (2). 

4, Information derived 
from 1,2,3, and 
from conduct of the 
work* 



(b) fidelity to objec* 
tlves 

(c) rate of performance 



(b) higher manage'- 
ment 

(c) scientifie 
community 



(d) Prospectiveness 

2, Syathesls of informa- 
tion for appropiiats 
levels of management 



Feedback 



Many R&D manaeers might conoider only phases I, II, and LI to be 
planning functions. Once the notion of a true planning cycle (one 
in which B&D outputs directly Influence future HCiD taskE is introduced, 
however, it becomes clear that all five phases muot be viewed as 
parts of a coherent whole. 
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A Model for the Evaluation of Management Planning Techniques 
J. vViUiam Smith and John L. Yeagerl 



With tlie passage of the Cooperative Research Act in 1954, there has 
been ao Inereaslng public Interest In the development of new educational 
programs throughout the naUon. In response to this interest, the Federal 
governmeiit has increased its allocatloii of resources for research and 
development in education. This larger pool of rasources has enabled the . 
reGearcher to mount slBnIflcantE6D Programs that transcend parochial 
coQcerns in order to address more critical and pervasive problems . As 
a direct result of this expanded support of research by the Federal 
government, larger and more complex research programs were formulated 
requiring management techniques not typioally found in the gener d field 
of education. Particularly, attentioti was given to the utilization of plan - 
nlng techniques in order to systematically study alternative actions in 
terms of costs and probablHty of success. At tlie present time, there is 
even grsater emphasis given to tlie plaQQlng of R&D Programs that have 
resulted from the plateaulng of Federal resources ooncommlt with an 
increasing public demand for the resolution of pressing edueatlonal 
problems. 

This concern for planmng has been evidenced by the ^.overnment in 
terms of the various planning studies it has funded over the past years , 
as exemplified by the support given to Cook's work in the development of 
an educational project management tha.ojy_ utilizing the Program Evaluation 
and Review Technique, Gephart's applioatlon of the Convergence 
Technique to research In reading, and the development of a Program-Plan- 
nlng-Budgeting-Evaluation Oystem by the Research Corporation of the 
Association of Sohool Buainesa Offloials. 

The implications are clear that planning activities will continue to 
attract significant interest from the Federal government, and consequently 
will exert a major force on the direction and scope of educational research 
activities. 



J. William Smith Is at the Learning Research and Development Conter, 
University of Pittsburg, Pittsburg, Pennsylvania, 15213 and 
John L. Yeager is at the Learning Research and Development Center 
University of Pittsburg, Pittsburg, Pennsylvania, 15213 
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Ihe purpose of this paper is to examine briefly the mechanics of some 
of the most cornmonly used planning techniques and provide a basis for 
selecting an appropriate technique for a reaearcher faced with limited 
financial resources who must aohleve a specific research objective within 
a given period of time. 

: I 

Towards this end, I will discuss Program-Plannlng-Budgetlng -Sys- 
tems md the Convergenoe Technique as techniques to be utilized for 
developing major research efforts eneompasslng many projects on a pro - 
grammatlc basis. Secondly, I will describe bar-oharting and the 
Program-EvaluatioQ-Eeview-Technlque as means of enhancing research 
maiiagement on a project level. The flaal part of this paper will deal 
with the development of a cross reference chart designed to assist the 
research maDoeer in the selection of a proper planning technique for his 
particular research effort. ' 

Program- Plaaning-Budgeting-Systams, generally referred to as 
PPB3, lends itself to the concept of a comprehensive plan because the 
technique requires the grouping of similar activities on a program basis 
and involves planning over an extended pcrfod of time, generally three 
to five years, Tlje primary purpose of PPBO is to provide a meehanlsm 
for the analysis of alteroatlves wiUun a framework of resource allocation. 

Operationally, the system requires the identification of research 
objectives and the development and analysis of programs aimed at 
attaining these objectives. The programs are then translated into bud - 
getary reqijirements reflecting all of tJie resources required for the 
sucessfulimp1.anv.. cation a given program. The program reqiured to 
meet a speolfic objective is the central element of tlie Program- Plannlng- 
Budgeting-System and as Cleland aad fCing have stateda the program is 
output - orieiited; It is defined in terms of what aohlevement is being 
sought, rather than what resources or Inputs. are available. 

A major problem facing the researcher In defining a program lies 
in the relationship bet^veen prog-rams and obJeoUvea. Besearch objectives 
are often ill defined, thus poorly stated in operational terms. Therefore 
the objectives, which serve as a basis for program definition, muot be 
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stated by the researcher in such a manner that the activities required by 
attainmetit of the objective can be identified and stated. Indeed, Fostsr 
has defined a progTam as "the sum of the steps or Interdependent activities 
which enter Into the attainment of a specified objective J' 

As the programs are preclaely defined and budgeted, their cost and 
benefits can be projected, tlius providing a cost-effsctiveness basis for 
evaluating alternative programs. Once this analysis has been accomplished 
and the major programs selected and budgeted, the plan is formalized by 
the preparation of a ''Program and Financial Pla^^ Cleland and King 
hcve defined the ^'Program and Financial Plan" as a statement of the 
program structure covering, on a year to year basis, a period of years 
determined by the nature of the organiEationjs objectives and operations, 
all within the format of a financial plan. This plan should reflect the 
optimum level of resources required to aohlevo tlia program objective- 
He=eval nation af changes in funding or changes in organizational compo - 
cents would, as the need bseame apparent, dictato any necessary changes 
in the projectad plan* 

In specific terms, a properly implemented PPB3 should, aeoordlng 
to Culbertsonj ^ f enable an organization toi 

1. '*Make available to top management mora 
concrete and speclflc data relevant to 
broad decisions; 

2* Spell out more concretely the objectives of 
programs; 



3, AnalyzB systematically and present for 
management review and decisicn possible 
alternative objectives and altar native 
progTams to meet tlieae objectives; 

4, Evaluate througlily and compare the 
henefita and costs of programs; 

5, Produce total rather than partial cost 
estimates of programs i 
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6. Present en a multi-year basis the proopective 
costs and accomplishmeati of programs; 

7, Review objectiyes and conduct program 
analysis oa a continuing, year round basis 

Instead of on a crowded schedule to meet ^ \ 
budget deadlltiss,^^ 

Perhaps a good llJustration of the purpose of PPBS in an educational 
environment has bsen shown in the Introductory statement of the PPE3 
Instruction Manual of fee University of Pittsburg, which In part states: 

"During the past several years, the UniverGity has 
been developing a Planning^ ProgTamming, Budaetlng 
System (P*P*B,3,) to provide a means for communl - 
cation within the University and a rational basis for 
planning and budgeting. Through such a system tlie 
various departments and areas will have a uniform 
means for communicating the scope of their progTams 
in terms of the resources required (finances j staff, 
spaoej etc.) while at the same time demonstrating 
whattliese programs produce (gTaduates, courses, 
research unlts^ publications, etc.)* A more clear 
understanding^of the specific programs which 
partments are providing, or proposing to provide ^ 
and the interrelationchips between these programs 
now becomes possible. Guch an understanding will 
facilitate the process of decision-making at all 
levels among all Interosted parties^-^facultys 
students, and administrators. By directly associating 
the budget v/ith specific programs and program levels, 
the actual pattern for use of Uclversity financial 
resources is made known and better informed 
budgetary decisions are possible." 

Clearly then, the advantage of PPBS lies prlmaryly in its ability 
to show the Interrelationship between program efforts and in the 
provision of budgetary statements which can bs used for analysis of 
alternatives. 



While PPB3 is well suited as a master plantiing techtilque for 
organJzatlona with many objectives, its usefulness to major researoh- 
orlented programs might be limited If the research la such that the 
solution to apartfcular Pi'obleni requires a wide range of activities 
aimed at the attainment of one particular objeotive. Typical of this 
type of research effort is the U.S. Office of Education's program of 
research on the problems of reading. An outgrowtli of this progTam was 
the examination of the Converganee Technique as a planning device for 
education. 

As Gephart relates, (5) that while personnel in several funding 
agencies had begun to explore means for mountlag a concentrated 
attack on the probleni of reading, there also existed a pessimism 
about the effectiveness of programmatic efforts because of the failure 
to generate program proposals which oould he oummulatcd for a 
problem solution. Beoause of this situation a decision vvas made to 
apply the Convergence Technique to basic studies In reading In an 
effort to develop a oomprehenslve, long-term reading program. 

Applications of the Convergenoe Techmque involves the formation 
of a planning team, usually oonaiating of BpeclaUst and generallst in a 
partioular field, as well as a systems aQalyst. This planQing team la 
responsible for defining the objeoiive and sub-objectives of the program. 
The sub-objectives are then sequenced so as to depict a logical move- • 
ment towards the major objecttve. The planning team is also reapon - 
sible for determining the ressarch activities required to achieve each 
sub-objective and establishing some criteria for tliis achlsvement. 

At the oonclusion of planiilng aotlvity, a dlagTam, or Conver - 
genes Cahrt, is coQstructed. This chart is then a bauis for deter - 
miiii,ng the projects to be undertajcen and the order in which they are to 
be started. Progress of the actlviMes Is then monitored and the plan is 
updated if necessary as tlie program progresses. 

Gephart(5) has exFlained tiie management fuiiotions of the Conver - 
gence Techuique in terms of examination of tlie project results against 
the relevant sub-objeotlve uriterla.' As tlie criteria for one sub-obieo- 
tlve Is satlafled, work towrrds the next sub-objective is hsgun. He 
fun ho? relates that If the orlterla for the gub-objectlve la not met, the 
logic of tlie program must be re-examined to determine if tJie plan 
should be changed or if additional worlc on the sub-obj?otive Is 



neeessary. TUs process la repeated until the program objective is 
accomplished. 

The basic advantage to research aotivltles of this technique Ues in 
the asaumptloQ that not all neoessary aotlvlttefl can be clearly identified 
m advance. The technique recognizes that research efforts Involve 
numerous unlmown factors. The logic is that as progress Is made the 
number of unknowns should decrease and the research efforts converge 
on a single solution. This Is best Illustrated by reviewing an example 
of a Convergence Chart. If you examine Figure I of your hand-out, 
which is a tyiaoal example of an over-all convergence chart, you can 
see the flow of aotivltlea and the arrays which depict the activities of the 
Planmng team as well as tlie meohanlams for monitoring resource 
allocation and Information flow. 

The major segments of the chart are the aeaearoh Flow, the 
Information Flow, and the Resource Flow. The Research Plow is 
usually subdivided into Linear, Concurrent, and Supplementary Arrays. 

Of. the arrays, the main research effort, represented by the Linear 
Array, is subdivided Into an appropriate number of flow patterns to 
mdlcate the major research sub-objectives that have been defined and 
to show how tlielr completion should converge the program's progress 
to flis attainment of tlie over-all obejctive. The Concurrent and 
Supplementary Arrays depict consideration of aotivltles not essential 
to the program, but whioh might have a considerable effect if success - 
fully completed. 

The Information Flow represents communication activities between 
program elements that are necessary to reflect the ongoing research 
efforts. 

Resource allocation and utilization monitoring is acoompUshed by 
developing and updating the Beaource Flow. A major advantage 
obtained from the appUoatlon of Convergenee Planning to research 
efforts lies In the loglo of systematic consideration of sub-objectives , 
the oompletlon of which leads to solution of toe research problem . ' 

Botii PPBS and the Convergence Teohnlque are suited for major 
large-scale, complex studies involving several groups and for research 
efforts expected to last over some eirtended period of time. These 
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planning devices are also appropriate for applicatloiiG where specific task 
are not necessarily required to be completed by some predetermined 
exact data. 

Obviously, a researther, vvho is planuiiiL a -iOro modest research 
effort, would find these two technlque.3 iiir iproprlate to his need due to 
the complexity of the tschnlques them:5etvaa. Assuming that all the 
oomponents of the purposed research are kno^vn, the concepts of project 
management^as exemplified by Project Planning Charts and the Progiain- 
Svaiuatlon-Review-Technique could then be utilized. 

Project Planning Charts, or bar charts, are an outgrowth cf 
Gantt Charts. Charts of thiD nature typically consist of a scale divided 
into units of time across ths top and project activities listed down the 
left-hand side. Figure II of your hand-out ahowc a typical example of a 
Planning Chart* Notice that bars or lines are dra\vn across the chart to 
indicate the schedule of each activity. As proorGCs is made on any -Iven 
activity, a second line Is drav/n in to show the status of the aQtlvlty in 
relation to the time scale. Hesouree allocations can then be made based 
on tlie shortest reasonable time needed to complete tlie project. 

Preparation of this type of chart requires tlid determination of a 
speciflo oourse of acUon to be follov;ed in the accomplishmont of the 
project. The course of action should be stated in terms of the necessary 
activities to be scheduled. Time estimates for each activity should be 
obtained from the person rGsponsible for tliat activity. Once tliis infor- 
mation is gathered, the chart can be prepared. 

Charts of this nature offer the advantage of showing the overall plan, 
the estimated schedule, and the current progreGs of the project 
simultaneously. 

Cleland and IClng t^) ^^ve identified the disadvantages of this technique 
principally as having to plan and schedule at the same time. Since tlie 
technique is based on a ttaie factor, alternative plans generally have to be 
evaluated on the basis of tlie initial schedule, tiius minimizing the oppor - 
tunlty to consider plana requiring a different schedule. In addition, bar 
ohMts do not show tlie Interdependent relatlonihlps of one activity to 
another. Although the technique will show tt^j amount of time that a given 
activity is behind schedule, It does not clearly show the Impact upoii 
subsequent activities or concurrent aotlvitles* 



These Project ProgTess Charts are best suited for reasonably small, 
short-term projeots which can be easily managed by monitoring the prcgreag 
of soma major activities and communicating the overall status of the 
project. 

Many of the shortcomings of progress charts are overcome by the 
Program-Evaluation-Hevlevv-Teohnlque. Commonly referred to as 
PEHT, this planning device has been Investigated and tested by Coolc for 
Its approprjPtenesfl to project management efforts In education. Accottllng 
to noman, PERT is designed to- 

"focus managerial attention on key project 
developmental aotivities; point up potential 
problem areas which could diarupt project 
objectives; evaluate progress towards the 
attainment of the project objectives; give 
management a prompt meehanloal reportitig 
device; and, aid and facilitate decision 
makir.g, " 

To aocompltsh these managerial objectives PERf utilizes time as a 
common denominator to reflect the Interrelationships of time, resource 
allocation, and performance standards. 

The first step In the Implementation of PERT is the definition of the 
project objectives. These objectives are then subdivided into identifiable, 
easily-managed activities which are classified according to their inter - 
dependeBcy relationships. As Cook has shown, ("^the subdividing and 
classifloatlon of activities continue until an appropriate level of detail 
is reached. The dependency relationshlpa existing between aotivities 
can then be deplated by use of a network systeni. 

Figure m shows that networks consist of circles and arrows (or 
appropriate substitutes) . The circles represent events and the arrows 
activitios. An event is apecific instant in time and consumes neither 
tim.por resources. An activity is actual task or work and consumea 
time and resources. 
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The specific requirements and characteristics of network systems 
Include- 

1. The individual tasks rsquired to achieve some objective must be 
defined in sufficl^.nt . detail so as to be represented in the network, which 
IS comprised of events aad activities. An event represents a spDclfic 
accompllshmeDt at a parH cular inatant In time. An activity represents the 
various resources, including time, whloh must be utilized to progress 
from one event to the next. 

2. Events must be sequenced on tlie network In such a manner as to 
allow identification of critical paths, which deplats tliat chain of necessary ■ 
events that in sumraation reqiiires the greatest time. 

. 3. Three time estimates are made for each activity of the network . 
These three time estimates are known as the most likely time, the 
optimistic time, and the pessimlstio time. The most,Uicely time is that 
time which, m^.the estimator's opinion, the activity vd^' require under 
normal conditions. The optlmlitlc time is that time required under the 
most favorable conditions . I'he pesslmlfitic time is that time required 
if just about everything goes wrong. These three time estimates provide 
a basis for calculating the expected time and the scheduled time of a 
project. The differenoe between these estimates can be expressed m terms 
oi a standard deviation and provide a probability of accomplishing 
scheduled time by use of -lie normal distribution table. 

^ In using network systsms, provisions must be made for frequent 
remew of progress as compared to the time estimates originally given for 
the various activities. Blnce the time predicticns are then compared with 
feedback in the form of performance data, opportunities are presented to 
re-allocate resources if neceasary to faoilltate progress along critical 
paths. 

Network systems, then provide a large amount of Information 
relating to activities neoesaary to aeoompllsh some objective to be pre - 
sented in a highly ordered fasliion. 

By estimating the cost of each activity, it is possible to add a 
dimension of resource oontrol to network systems, thus providlr.g a better 



- 80 - 



basis for manage meat action. 



Network systems can be applied to almost any project where lorical 
planning is required. Cook Identifles several criteria that should be 
conrndered in the deciBion to utilize network systems. 

1, '*Does a tjpecifled end objective exist, the 
accomplislimeiit of which can be determined 

2, '^Must some scheduled date or deadlina be 
met?'* 

3* ^'What is the degree of project complexity?'^ 

4. ^^Does a degree of unceminty exist as to 
the definition of some or all program 
elements 

5, ''How familiar are the concepts of PEHT to 
project personnel?*' 

A prlmaiT disadvantage of utilizing PERT for research activities is 
the lack of ability to specify with great accuracy the time elements of 

^^"''"'^^ if toe elements are properly estimated 

au^ weakness need not mvalldate the effectlveueas of PEBT to educational 



The description of thsse techiilques has been designed to state only 
the basic prmclples of each method and to provide some Insight for its 
appropriate applications to the field of eduoaUonal research." 

Figure W Is a selection criteria chart designed to assist the 
researcher m selecting an appropriate planning device for his particular 
needs. A review of this chart will Indicate some considerations that ■ 
should be taken and the consequences that would result from the selection 
of any of these techniques. 

The criterion listed in this chart and the corresponding ratings are 
not to be interpreted as absolute values for all research situations, but 
rather as an indication of the appropriateness of the technique for a 
typica]" situation. The criteria represents an initial attempt to identify 
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selected characteristics that Indicate svhether or not a particular techiilciiH: 
should be applied to a given research activity. 



It has been my purpose in tliis paper to identify the basic features of 
four techniques that can and have been used in educational endeavors and 
to relate some meaningful criteria that will assist educational researchers 
in the selection of an appropriate planning device. It is anticipated that as 
the general area of planning receives increasing attention by the educational 
community, new planning techniques and criteria will evolve. This will 
permit the researcher to more effectively conduct his research activities. 
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FIGURE II 
PRCJ2CT PLANNINC CHAK'i 
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FIGURE m 
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.jELSCTION ciutehi.\ 

Key 1 = Low appllcaticn of statement to tochp'nue 

2 = Moderate application of si tenient to tachnlque 

3 = High application of statement to toclinique 

I. The Je c hniQue is best suited for; Bar -ch arts PERT 

1 - Complex multi-organization 

research effort 1 £ 



1 

1 



2 - Single objective-project resrarnh 2 

3 - Basic research (large circle) 1 

4 - Basic research (modest scale) l 

5 - Developmetital aspects of 

educational research 2 

6 - Identifying and analyzing 

alternative strategies 

7 - Costing alternative strategies 
II. T he tech nig ue requirgs! 
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II- I he technigue requires : (cont^d) : Ear-charts PERT Conve rgence P]P \ 



6 = Establishment of a defined 

schedule 3 

7 ^ Identify of specific activities 3 

8 = Extra cost for implementation of 

technique to re^isarch effort 1 

9 - Detailed specification of all 

component activities 3 
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